The state-of-the-art video coding standard, such as H.264/AVC (Advanced Video Coding) and H.265/HEVC (High Efficiency Video Coding) standard, is based on a hybrid coding scheme using block-based prediction and transform coding. First, the input signal is split into rectangular blocks and then predicted from previously decoded data either by inter prediction or intra prediction. The resulting prediction error is coded by applying block transforms and quantization.
In a coding scheme that uses block-based prediction and transform coding, visual artifacts can occur in reconstructed pixels at the block boundaries. Visible discontinuities at the block boundaries are known as blocking artifacts, and after block-based compression, visual artifacts can occur in the reconstructed pixels. A deblocking filter has been designed to eliminate these artifacts and improve the subjective quality, and adopted in many video coding standards, such as H.264/AVC and H.265/HEVC standards in which a frame is divided into several square coding units (CU). A difficulty associated with designing an appropriate deblocking filter is to determine whether to filter a particular block boundary, and if the deblocking filter were to be applied, then to determine on a filtering strength.
In the H.265/HEVC, one of key features of the H.265/HEVC structure is that it has the multiple partition conceptions. A frame is split into several coding tree units (CTUs), and a CTU is then split into CUs by using a quadtree structure denoted as coding tree to adapt to various local characteristics. Each CU can be further split into one, two or four prediction units (PUs) according to the PU splitting type. Inside one PU, the same prediction process is applied and the relevant information is transmitted to a decoder on a PU basis. After obtaining the residual block by applying a prediction process based on the PU splitting type, a CU can be partitioned into transform units (TUs) according to another quadtree structure similar to the coding tree for the CU.
A new block structure, quadtree plus binary tree (QTBT), has been proposed for the next generation video coding standard beyond H.265/HEVC. The QTBT structure removes the concepts of multiple partition types, i.e., it removes the separation of the CU, PU and TU concepts, and supports more flexibility for CU partition shapes. In the QTBT structure, a CU can have either a square or rectangular shape. A CTU is first partitioned by a quadtree structure, and the quadtree leaf nodes are further partitioned by a binary tree structure. In the binary tree splitting, there are two splitting types, symmetric horizontal splitting and symmetric vertical splitting. The binary tree leaf nodes are called coding units (CUs), and the binary tree segmentation is used for prediction and transforms processing without any further partitioning. This means that the CU, PU and TU have the same block size in the QTBT coding block structure.
In the H.265/HEVC, there are only square CUs and the H.265/HEVC deblocking filter is designed for square CUs. However, the QTBT introduces non-square CUs, and the deblocking filter designed for the square CUs may not provide efficient compression performance for non-square CUs.